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Abstract. A poset has the non-messing-up property if it has two covering sets of disjoint 
saturated chains so that for any labeling of the poset, sorting the labels along one set of 
chains and then sorting the labels along the other set yields a linear extension of the poset. 
The linear extension yielded by thus twice sorting a labeled non-messing-up poset may be 
independent of which sort was performed first. Here we characterize such sort-invariant 
labelings for convex subposets of a cylinder. They are completely determined by avoidance 
of a particular subpattern: a diamond of four elements whose smallest two labels appear 
at opposite points. 



1. Introduction 

The non-messing-up property of matrices is a well-known result about sorting the entries 
in a matrix. It appears in work by Boerner p], Gale and Karp [2], Knuth [3], and Winkler 
[6], and a detailed discussion of its provenance occurs in [5]. Given a matrix M, let C(M) 
be the matrix obtained by ordering the entries within each column of M, and likewise let 
7Z(M) be the matrix obtained by ordering the entries within each row of M. 
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Then the matrices C(M) and 1Z(M) are 
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Moreover, 
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Theorem 1.2 (Non-messing-up matrices). Consider a matrix M. The entries in each 
column in the matrix TZC(M) are in non- decreasing order; that is, TZC(M) = CTZC(M). 

Less formally, Theorem 11.21 says that after sorting the data within each column of a 
matrix, sorting the data within each row of the subsequent matrix does not "mess up" the 
fact that each column's data are sorted. Note that by symmetry, the entries in the rows of 
the matrix CTZ(M) must also be sorted, so CTZ(M) = TZCTZ(M) as well. 
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The expressions of ([I]) provide an example of the non-messing-up property of matrices. 
Note from these that it is not necessarily the case that the matrices TIC{M) and CTZ(M) 
coincide; that is, the sorting operations do not necessarily commute. It is this issue which 
we address here. Moreover, we examine it for more general non-messing-up posets, which 
we previously treated in [5]. 

In Section [2] we review the results of [5] and define the central object of this paper: a 
transverse non-messing-up poset. Our main result, stated precisely in Theorem 13.61 is that 
the sorting operations commute on a labeling of a transverse non-messing-up poset if and 
only if the labeling avoids a particular forbidden subpattern (subject to some degeneracies) 
consisting of a diamond of four elements whose smallest labels appear at opposite points. 
Unfortunately this result does not hold for non-transverse non-messing-up posets, as shown 
in Section^ The paper concludes with a brief discussion of sorted matrices and when they 
have a preferred sorting procedure; that is, when it is more likely that a sorted matrix arises 
from first column-sorting and then row-sorting, or vice versa. 

2. NON-MESSING-UP POSETS 

The goal of this paper is to describe exactly when the sorting operations 1Z and C commute 
on a labeling, and we do this in the context of non-messing-up posets. Previously, in [5], 
we defined and characterized a generalization of the non-messing-up phenomenon to the 
setting of posets. That work is summarized here, and more details can be found in [5]. 

Generalizing to the realm of posets is natural because an r-by-c matrix resembles the 
poset r x c, a product of two chains. The rows and columns of the matrix correspond 
to two sets of disjoint saturated chains covering the elements of r x c. Sorting a chain 
means putting that chain's labels in order so that the minimum element in the chain has 
the minimum label. Thus the non-messing-up property indicates that sorting the labels 
along both the rows and the columns gives a linear extension of r x c. 

Definition 2.1. Let P be a finite poset. A gridwork for P is a pair (R,C) where R and 
C are each sets of disjoint saturated chains covering the elements of P, and where each 
covering relation in P is contained in an element of R U C. We call the elements of R the 
rows of P, and the elements of C the columns of P. 

Definition 2.2. Let P be a finite poset with gridwork (R,C). The gridwork is transverse 
if each row and each column intersect at most once. 

Definition 2.3. Let P be a finite poset with a transverse gridwork (R, C). Given a labeling 
of the elements of P, let 1Z be the operation which sorts the labels within each row of the 
poset, so that the minimum element in a row has the minimum label of that row. Likewise, 
let C be the operation which sorts the labels within each column of the poset. 

We are now able to carry the notion of non-messing-up to the setting of posets. 

Definition 2.4. A poset P has the transverse non-messing-up property, or is transverse 
non-messing-up, if there exists a transverse gridwork such that, for any labeling A of the 
elements of P, both 1ZC(X) and CTZ(X) yield linear extensions of P; that is, TZC(X) = CTZC(X) 
and CTZ(X) = KCK(X). 

Before stating the characterization of transverse non-messing-up posets found in [5], we 
must state one more definition. 
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Definition 2.5. Fix positive integers k and n so that k < n. The cylinder poset Cylt y n 
is 7?/(—k,n — k)TL. The partial order on Cyl^^n is induced by the componentwise partial 
order on 1? . 

Theorem 2.6 ([5]). A poset is transverse non-messing-up if and only if each of its connected 
components P is a convex subposet of a cylinder poset. 

The gridwork for a non-messing-up poset is specified in [5j, and the sets R and C are anal- 
ogous to the rows and columns of the motivating matrix example. The reader is encouraged 
to find the details in [S]. 

Note that one can also consider non-transverse non-messing-up posets, and those are 
also classified in [5]. Such posets differ from transverse non-messing-up posets by allowing a 
row and a column to intersect more than once. Each non-transverse non-messing-up poset 
is obtained from a transverse one by replacing one or more elements by chains, subject 
to length restrictions. Not only does a non-transverse non-messing-up poset look quite 
different from the motivating matrix situation, but there is some redundancy in the sorting 
operations since, for example, chains of elements in the columns will already have been 
sorted by 1Z. 

3. Transverse sort-invariance 

Throughout this section, let P be a transverse non-messing-up poset with transverse 
gridwork (R, C). For the sake of clarity, suppose that in any labeling of P, the elements are 
labeled by {1, 2, . . . , \P\}. 

Definition 3.1. A labeling A of the poset P is sort-invariant if TZC(X) = CTZ(X). 

In other words, a labeling A is sort-invariant if the operations TZ and C commute on A. 
Our goal is to characterize sort-invariant labelings of transverse non-messing-up posets. 

Definition 3.2. Let x and y be elements of non-messing-up poset, that are in neither the 
same row nor the same column. Suppose that x is in row v x and column c Xj while y is in 
row r y and column c y (thus r x ^ r y and c x ^ c y ). The corner-set of x and y is the subset 

0(x,y) = (r x Ur y )n(c x Uc y ) (2) 

= (r x n c x ) U (r y n c y ) U (r x n c y ) U (c x n r y ). 

If x and y are in either the same row or the same column, then set §(x, y) = 0. 

Because P is transverse, if x and y are in neither the same row nor the same column in 
P, then we have 

0(x, y) = {x, y} U (r x n c y ) U (c x n r y ), (3) 
and this §(x, y) contains at most four elements. 

The nomenclature here refers to the fact that the set ()(x,y) looks like the four corners 
of a diamond containing x and y in P, when the edges of the diamond must be either rows 
or columns in the poset. Note that because non-messing-up posets (and their rows and 
columns) arise from convex subposets of cylinder posets, if x and y are comparable in P, 
then both r x n c y and c x PI r y are nonempty. 

In the case of the product of two chains, we will show that a labeling is sort-invariant 
if and only if corner-sets with extreme values on opposite sides are avoided. In general, a 
corner-set may have fewer than four elements, and we must avoid more particular patterns 
with these smaller sets, as stated in the second and third points of Definition 13.31 
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Definition 3.3. A nonempty corner-set fy(x, y) is bad for A (or simply bad if A is clear from 
the context) if \§(x, y)\ > 2 and one of the following situations is satisfied: 

• 0(x,y) = {x,y,w,z} and 

A(x), X(y) > AO), X(z) or A(x), \{y) < X(w), X(z), 

• 0(x, y) = {x, y, z} with x,y < z and X(x), X(y) > X(z), or 

• ()(x, y) = {x, y, z} with x,y > z and X(x), X(y) < X(z). 

If (}(x, y) is not bad for A, then it is good for X (or simply good if A is clear from the context). 
Example 3.4. The following four posets and labelings have only bad corner-sets. 
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3 1 




Example 3.5. The following six posets and labelings have only good corner-sets. 
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We are now able to state the main result of this paper, which characterizes exactly when 
a labeling of a transverse non-messing-up poset is sort-invariant in terms of its corner-sets. 

Theorem 3.6. Let P be a transverse non-messing-up poset and X a labeling of P. The 
labeling X is sort-invariant if and only if each nonempty corner-set in P is good. 

In other words, sort-invariance of a transverse non-messing-up poset (that is, a convex 
subposet of a cylinder poset) is characterized by a forbidden subpattern: a bad corner-set. 

Theorem 13.61 is proved in three steps, two of which we highlight as propositions. The 
first step is to analyze sort-invariance for the product of two chains (the poset version of 
the matrix), the second step is to consider convex subposets of the product of two chains, 
and the final step is to look at the cylinder. 

In the next proposition, we describe sort-invariance for posets which are the product of 
two chains. 
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Proposition 3.7. Let the poset P be the product of two chains, and let X be a labeling of 
P. The labeling X is sort- invariant if and only if each nonempty corner-set in P is good. 

Proof. First, note that P is a transverse non-messing-up poset. Let R and C be the rows 
and columns of P, which are analogous to the rows and columns in a matrix. The row which 
intersects each column in the minimum position of the column will be called the "first" row, 
the row intersecting in the position above that will be the "second" row, and so on. We 
similarly name the columns. 

We first note that sort-invariance is equivalent to saying that for all k € {1, . . . , \P\}, 
the shape formed by {1, . . . , k} in TZC(X) is the same as the shape formed by {1, . . . , k} in 
CTZ(X). Thus we can replace the labels in A by just {1^, 2^}, where lfc replaces each element 
of {1, . . . , k} and 2^ replaces each element of {k + 1, . . . , |P|}, and look at the resulting 
placements of the l^s after 7ZC and CTZ, with the understanding that 2^ > 1^. 

For the remainder of the proof we will drop the subscript k and just look at a labeling 
by Is and 2s, and decide when TZ and C commute on this filling. 

The operation C pushes all the Is to the bottom of each column. One interpretation of 
TZC is that it sorts the set of columns so that the column in which the most Is appears 
is now the first column, the column in which the next most Is appears is now the second 
column, and so on. The shape of Is resulting from CTZ can be described analogously. 

For these shapes to be equal, we need the ith most Is appearing in any row to equal the 
number of columns containing at least i Is. This is equivalent to the following criterion: let 
Sc C R be the set of rows in which column c has a 1; index the columns Ci, C3, . . . so that 
Cj_|_i has at least as many Is as Cj does; then we must have 

S Cl C S C2 C . . . . (4) 

(Equivalently, the roles of "row" and "column" can be swapped in the previous discussion.) 

For example, the following is a labeling satisfying (J3]), where the poset has been rotated 
to highlight the rows and the columns. 
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The Cj designations are given below each column. Note that c\ and can be swapped. 

We claim that the inclusions of (j3]) are equivalent to having no bad corner-set. Certainly 
if the inclusions of hold then there is no such bad corner-set. Conversely, if there is no 
bad corner-set then consider any pair of columns c and d , where c has at least as many Is 
as d . If there is a row in which c has a 1 but d does not, then every row in which d has a 
1 must be a row in which c does as well. This implies ([!]), and completes the proof. □ 

Proposition 3.8. Let the poset P be a convex subposet of the product of two chains, and let 
X be a labeling of P. The labeling X is sort-invariant if and only if each nonempty corner-set 
in P is good. 

Proof. Suppose P is a convex subposet of P', where P' is a product of two chains. Extend 
the labeling A of P to a labeling A' of P' by giving the label 1 to every element of P' \ P 
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less than an element of P, and the label |P| to all other elements of P' \ P. Clearly A is 
sort-invariant if and only if A' is sort-invariant. Similarly, A has a bad corner-set if and 
only if A' has a bad corner-set. Proposition 13.71 links these two concepts for A', and thus we 
conclude that A is sort-invariant if and only if all of its non-empty corner-sets are good. □ 

Proof of Theorem \3.6[ A labeling of a poset is sort-invariant if and only if the labeling of 
each connected component of the poset is sort-invariant. Thus we need only prove the result 
for a connected poset. The remainder of the proof resembles that of Theorem 6]. 

Let P be a finite convex subposet of Cylk, n - Cut Cylk, n to get a preimage of P in the 
plane. Glue together copies of this poset via the identifications on the cylinder. After 
perhaps removing elements at the sides of the planar poset, this is a convex subposet of a 
product of two chains. For the labeling A of P, label every preimage of x in the plane by A(x). 
Glue enough copies of the poset so that after the two sorting procedures, sufficiently many 
of the centermost copies in the plane have the labels they would have had on the cylinder. 
This is possible because only finitely many elements cross over a line of identification. Call 
this glued-together poset P', and its labeling A'. 

The labeling A has a bad corner-set if and only if A' does. Thus, by Proposition l3.8l A has 
a bad corner-set if and only if A' is not sort-invariant. If A' is sort-invariant then certainly 
A is as well. Therefore, if A has no bad corner-sets, then A is sort-invariant. 

On the other hand, suppose that A has a bad corner-set. Thus A' has a bad corner-set, 
and is not sort-invariant. We must be sure that the sort-invariance fails somewhere in one 
of the centermost copies of the cut poset. This will ensure that A is also not sort-invariant, 
which will conclude the proof. Since Proposition 13.71 characterizes sort-invariance in the 
plane by forbidding bad corner-sets, interacting with the labels appearing in a bad corner- 
set during 1Z or C must lead to the failure of the sort-invariance. By construction of P', such 
failure must appear among the centermost copies of the cut poset. Since these centermost 
copies have the same labels that they would have had on the cylinder, this means that A is 
not sort-invariant. □ 



One might hope that sort-invariance for non-transverse non-messing-up posets can be 
characterized by an analogous statement about bad corner-sets. Unfortunately the obvious 
generalization of this concept is not the answer. More precisely, recall the definition of 
a corner-set, and in particular equation ([2|). In a non-transverse poset, the subsequent 
equation ([3]) does not necessarily hold, and the corner-set might well have more than four 
elements. We can extend the definition of "bad" in the obvious way, to say that a corner-set 
is bad if there exist elements 
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However, avoidance of such a bad corner-set is not enough to guarantee sort-invariance in a 
general (non-transverse) non-messing-up poset, as demonstrated in the following example. 

Example 4.1. Figure [1] gives a non-transverse non-messing-up poset P (as described in 
[5]) and a labeling A of P. 
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Figure 1. A non-messing-up poset P whose rows are drawn with dashed 
lines and columns with solid. A labeling A of P is also given. 

Notice that, in the sense described at the beginning of this section, the labeling A is good. 
However, it is not sort-invariant, as demonstrated in Figured 
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Figure 2. The left-hand poset is 1ZC(A), and the right-hand poset is CTZ(X). 
They are unequal, and thus A is not sort-invariant. 



It is disappointing that the equivalence between sort-invariance and avoidance of a bad 
corner-set does not carry over to this more general setting, and it remains open to charac- 
terize sort-invariance of these non-transverse non-messing-up posets. 

5. Preferred sorting procedures for matrices 

We conclude with a brief discussion of a different but related question regarding sorted 
matrices; namely, whether a given sorted matrix A is more likely to have arisen from 1ZC or 
from C1Z. Certainly there is a matrix M which can yield A under both of these operations, 
namely M = A, but there may be a significant preference for 1ZC, say, in general. 

Say that the matrix is sorted if the data in each row are non-decreasing from left to 
right, and if the data in each column are non-decreasing from top to bottom. Note that the 
transpose of a sorted matrix is also necessary sorted. Given a sorted matrix A, there are 
many possible original matrices M with TZC(M) = A or with CTZ(M) = A. Furthermore, 
the set of such M depends on whether the rows were sorted first and then the columns, or 
vice versa. For a given sorted matrix, it may be that one of these options is preferred to 
the other. 
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Without loss of generality, suppose that the data in the matrix is the set {1, . . . , rc}, 
where the matrix has r rows and c columns. Using [4, Exercise 7.20b], we can compute the 
number of matrices M for which 1ZC(M) = A, and likewise for which CTZ(M) = A. First 
we define the function 1%a on the set {1, . . . , rc} of entries in A: if i is in the first row of A 
then tiA(i) = 1; otherwise, 

Ha {i) = #{i < i '■ J 'is in the row immediately above i in A and not to the left of i}. 

Set 

h A (A) = JJM0- 

It follows from [H Exercise 7.20b] that the number of matrices M for which TZC(M) = A is 

MA) • (H) c • cl 

Likewise, the number of M for which CTZ(M) = A is 

^ T (A)-(c!y-H, 

where ^4 T is the (sorted) transpose of the sorted matrix A. 

Therefore we can make the following conclusion about preferred sorting procedures. 

Proposition 5.1. Suppose that A is a sorted r-by-c matrix. Let M be an r-by-c matrix 
chosen at random, and flip a fair coin to determine whether to sort M by performing 1ZC 
or C1Z. Then the probability that A = TZC(M) is 

h A (A) • (riy- 1 

h A {A) ■ (H)e- 1 + h A T(A) • ( c rr-i- 
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